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Description 



[0001] The present invention relates to a sign language translation system and method for recognizing a sign lan- 
guage and translating it into a spoken language. 

[0002] As described in Proceedings of Conference on Human Factors in Computing Systems CHI'91 (1991), pp. 237 
to 242, a conventional sign language translation method recognizes a sign language by converting the motion of hands 
wearing gloves with special sensors into electrical signals and checking whether the pattern of signals matches any 
one of previously registered signal patterns. 

[0003] Another conventional method described in JP-A-2-1 44675 recognizes finger spellings by taking an image of 
a pair of colored gloves by a TV camera to derive color information of differently colored areas of the gloves and by 
checking the pattern of the derived color information whether it matches any one of previously registered color infor- 
mation patterns. 

[0004] There are three types of Japanese sign languages, including (1) "a traditional type sign language" having 
been used by auditory- impaired persons, (2) 'a simultaneous type sign language" in which each sign language word 
is assigned a Japanese word, and (3) "an intermediate type sign language" in which each sign language word is 
arranged in the same order as a spoken language. The intermediate type sign language is most frequently used. With 
this language, sign words are given in the same order as a spoken language. Consider for example a Japanese sen- 
tence " V»« ^i»tt- Sr=> B equivalent to a romanize sentence 

"Watakushi wa fuyu ni hokkaido e ikou toomou" and corresponding to an English sentence "I am thinking of going to 
Hokkaido in the winter". In this case, only important words (also called independent words) such as a verb "»< "= "iku" 
= "go" and nouns **c*iL u = "hokkaido" (Japanese district name), and 0 = "fuyu" = "winter", are expressed in the 
sign language, and other words are generally omitted such as postpositions "t. c = "te, ni, wo, ha" = "various words 
or word elements similar to English particles but postpositional according to the Japanese Grammar", words dependent 
on auxiliary verbs " i*. *»> " = "reru, rareru, you, tap = "various words or word elements used in a corresponding 
manner in English), and pseudo nouns "e « " = "koto, mono, no" = "various words used in a corresponding manner 
in English". In addition, the conjugations of conjugative words such as verbs, auxiliary verbs, adjectives, adjective 
verbs, are generally omitted. Accordingly, the first -mentioned conventional method which simply arranges sign lan- 
guage words in the order recognized, is difficult to express a spoken language. 

[0005] The sign language mainly used in the U.S.A. is the American Sign Language (ASL). In the U.S.A. the "tradi- 
tional type sign language" is widely accepted. Also in the case of ASL, articles such as "a, an, the" and prepositions 
such as "in, at, on, of, by, from" are often omitted. 

[0006] The second-mentioned conventional method recognizes Japanese finger spellings °j> " = "a", % " = V, "> " = 
"u" and so on, corresponding to English alphabets. Finger spellings are used as an alternative means for the case 
when sign words cannot be remembered or understood, and so conversation using only finger spellings is rarely had. 
This conventional method is not satisfactory for conversation, because each word in a sentence is expressed by giving 
the finger spellings of ail characters of the word. 

[0007] Such problems of sign language recognition are associated not only with Japanese sign languages but also 

with other sign languages for English and other foreign languages. Such problems are inherent to the sign language 

which provides communications by changing the positions and directions of hands (inclusive of fingers, backs, palms), 

elbows, and arms, and by using the whole body including the face, chest, abdomen, and the iike. 

[0008] According to a first aspect of the present invention there is provided a sign language translation system, 

comprising: 



input means for inputting the motion of hands as electric signals; 

language generating means for generating a spoken language from said inputs, including sign language word 
generating means for recognizing words corresponding to hand motions in accordance with said inputs and gen- 
erating sign language words; 

output means for outputting said generated spoken language, 

characterised in that said language generating means further includes: 

storage means for storing conjugations of those of said sign language words which are conjugative words, and 
omitted words to be supplemented between said sign language words, said omitted words being words not cor- 
responding to hand motions; 

dependence analysing means for analysing a dependence relationship between successive ones of said recog- 
nized words and outputting analysed results; 

spoken language generating means for generating said spoken language in accordance with said analysed results 
by providing said stored conjugations of said conjugative words and supplementing said stored omitted words 
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between said recognized words. 

[0009] According to a second aspect of the present application there is provided a sign language translation method 
comprising the steps of: 

inputting the motion of hands; 

recognizing words corresponding to said inputted motion of hands as sign language words; 
generating a spoken language using the dependence between said recognised words; and 
outputting said generated spoken language; 

characterised in that: 



conjugations of those of said sign language words which are conjugative words, and omitted words to be supple- 
mented between said sign language words, said omitted words being words not corresponding to hand motions 
is are stored; 

the dependence between said recognised words is analysed, and a predetermined process is executed using said 
analysed results to generate said spoken language by supplementing said omitted words and providing said conjuga- 
tions of words that require conjugation. 
20 [0010] Thus, case postposition (Japanese postpositional particle) are supplemented between words having a de- 
pendence relationship: a case postposition determined by the dependence relationship is inserted between the rec- 
ognized words. 

[001 1] In addition, the omitted conjunction at the ending of each conjugative word having a dependence relationship 
is determined, while referring to the predefined conjugations of conjugative words including verbs, adjectives, adjective 
25 verbs, and auxiliary verbs. 

[0012] Also supplemented are omitted auxiliary verbs, postpositions, pseudo nouns, and conjunctions, respectively 
providing the semantic and time sequential relationships between depending and depended words having a depend- 
ence relationship. 

[001 3] For example, given a train of words in a sign language "^ " = "watakushi" = V, " = "fuyu" = "winter", °*3^1 ■ 
30 = "Hokkaido", " ft< " = "iku" = "go", and"t*V = "omou" = "think", the dependence analysis finds that the words ■•fas', 
' and "4t3Rl" are the subjective case, place case, and time case of the depended verb "ft< ", respectively. Next, 
the omission/ supplement rule supplements case postpositions "wa", "ni", and "e" after the words B ^°, and 
"> respectively. A case postposition "t " = "to" = "of" is added before the verb 'Ts'i ". An auxiliary verb ■» " = "u" 
(meaning an intention) is added between the words "tf < " and ". A conjugation at the ending of the conjugative word 
35 -«< - - "iku" = « go " j s changed in order to correctly connect, from the viewpoint of grammar, the words "ft< ",">", and 
Sro " = "to omou" = "thinking of". As a result, a spoken language sentence " 4m* £i= itts-^ « 21 * " = "Wa- 
takushi wa fuyu ni hokkaido e ikou to omou" = "I am thinking of going to Hokkaido in the winter" can be generated. 
[0014] In the drawings: 

40 Fig. 1 shows the structure of a sign language translation system according to an embodiment of the present in- 

vention; 

Fig. 2 is a diagram illustrating the outline of the operation to be executed by the embodiment system; 
Fig. 3 shows the details of a word recognition procedure; 
Fig. 4 is a diagram illustrating a dependence analysis procedure; 
45 Fig. 5 shows the details of the dependence analysis procedure; 

Fig. 6 shows the details of an omission/supplement procedure; 
Fig. 7 shows the data structure of recognized sign data; 
Fig. 8 shows the data structure of an analysis table; 
Fig. 9 shows an example of a word dictionary; 
50 Fig. 10 shows the structure of an analysis stack; 

Fig. 11 shows an example of a case dictionary; 
Fig. 12 shows examples of omission/supplement rules; 
Fig. 1 3 shows an example of a conjugation table; and 
Fig. 14 shows an example of a generated sentence. 



[0015] Embodiments of the present invention will be described with reference to Figs. 1 to 14. 

[0016] Fig. 1 shows the structure of a Japanese sign language translation system according to an embodiment of 

the present invention. This system is realized by a computer having a CPU 102, input devices (keyboard 101 and sign 
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data input device 1 05), an output device (a CRT 1 03), and a memory 1 04. Stored in the memory 1 04 at a predetermined 
memory area are programs 2 for generating sentences of a spoken language expression from inputted sign data 7 
supplied from the sign data input device 105. The programs 2 are loaded to the CPU 102 for various procedures to be 
performed. These programs 2 include a program 3 for recognizing a train of words from the inputted sign data 7, a 
program 4 for analyzing dependency relationship between words, and a program 6 for supplementing omitted words 
in and between words having the dependency relation to generate a sentence 14. Reference numeral 7 represents a 
memory area in which the inputted sign data is stored. Reference numeral 8 represents a memory area in which an 
analysis table to be used by the programs 2 is stored. Reference numeral 10 represents a memory stack area to be 
used by the dependency analyzing program 4. Reference numeral 9 represents a memory area in which a word dic- 
tionary is stored, the word dictionary storing the meaning and part of speech of each word to be used by the sign 
language. Reference numeral 11 represents a memory area in which a case dictionary is stored, the case dictionary 
being used for defining the case of a word dependable on a predicate word such as a verb to be used by the sign 
language. Reference numeral 12 represents a memory area in which omission/supplement rules are stored, the rules 
providing a method of supplementing omitted words in and between words having the dependency relation. Reference 
numeral 13 represents a memory area in which a conjugation table is stored, the table defining the conjugation rule 
for conjugate words. Reference numeral 14 represents a memory area in which generated sentences are stored. 
[0017] In the sign language translation system shown in Fig. 1, CPU 102 generates translated sentences in accord- 
ance with the programs loaded from the memory 104. In this case, a discrete hardware structure may be used for the 
execution of part or the whole of the word recognition (corresponding to the program 3 in Fig. 1 ), dependency relation- 
ship analysis (corresponding to the program 4 in Fig. 1), word omission/supplement (corresponding to the program 6 
in Fig. 1), and other functions. 

[001 8] Fig. 7 shows the detailed structure of inputted sign data in the form of a table. This table stores the history of 
right and left hands data 71 and 72, which represent states of right and left hands, supplied from the input device 105 
at a predetermined time interval. It is well known to use gloves for the input device 105, which gloves allow the position 
and direction of each hand and the flection of each finger to be converted into electrical signals. The motion of hands 
may be inputted by using a TV camera for example. The right and left hands data 71 and 72 are stored in the table at 
the locations 73, 74,..., 75 at sampling times TO, T1 Tn, respectively Data 711 indicates the first articulation angle 
of a thumb of the right hand, and data 712 indicates the second articulation angle of a thumb of the right hand. Data 
71 3 indicates the x-coordinate position of the right hand, data 714 indicates the y-coordinate position of the right hand, 
and data 715 indicates the z-coordinate position of the right hand. Data 716 indicates the angle of the direction of the 
right hand relative to the x-axis, data 717 indicates the angle of the direction of the right hand relative to the y-axis, 
and data 718 indicates the angle of the direction of the right hand relative to the z-axis. 

[0019] Fig. 8 shows the details of the analysis table 8 which stores data of a train of words 88 obtained through 
analysis of the data 7 supplied from the input device 105. This table is constructed of a word field 81 , part of speech 
field 82, meaning field 83, depended word field 84, dependency relationship field 85, and supplemented result 86. 
[0020] Fig. 9 shows an example of the word dictionary 9 which stores a part of speech 92 of each word 91 used by 
the sign language and the meaning 93 thereof. 

[0021] Fig. 10 shows the structure of the analysis stack area 10, indicating that words 1001 to 1004 were stacked 
in this area during the dependency analysis. 

[0022] Fig. 11 shows an example of the case dictionary 11 which stores a dependable case 112 for a predicate word 
such as a verb 111, dependable case meaning 1 1 3, and case postposition 1 1 4 to be added to the word of the dependable 

case, 

[0023] Fig. 12 shows examples of the omission/ supplement rule 12. The rule 121 illustrates a supplement method 
123 of supplementing a depending word, when the condition 122 is met wherein all conditions 1221, 1222, and 1223 
are concurrently satisfied. Similarly, the rule 124 illustrates another supplement method 126 of supplementing a de- 
pending word, when the condition 125 is met wherein all conditions 1 251 to 1254 are satisfied. For example, with the 
rule 121, on the conditions that the part of speech of a depending word is a noun, pronoun, or proper noun, that the 
part of speech of the depended word is a verb, and that the dependable case is a subjective case, time case, position 
case, or objective case, a case postposition for the dependable case is supplemented to the depending word. 
[0024] Fig. 1 3 shows an example of the conjugation table 1 3 showing a methodical presentation of the conjugations 
of conjugative words such as verbs and auxiliary verbs. 

[0025] Fig. 14 shows an example of a generated sentence 14 subjected to supplement. 
[0026] The operation of the embodiment will be described with reference to Figs. 2 to 6. 

[0027] Fig. 2 is a diagram illustrating the overall operation to be executed by the embodiment system. Steps 4 and 
6 shown in Fig. 2 concerns the subject matter of the present invention. 

[0028] At Step 3, a train of words are recognized from the inputted sign data 7, and loaded in the analysis table 8. 
The details of Step 3 will be given later with reference to Fig. 3. 

[0029] At Step 4, the dependence relationship between the words of the recognized train is analyzed by using the 
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analysis table 8, word dictionary 9, analysis stack area 10, and case dictionary 11 . The analyzed results are stored in 
the analysis table 8. The details of Step 4 will be given later with reference to Figs. 4 and 5. 

[0030] At Step 6, omitted words between the words having the dependency relationship are estimated by using the 
analysis table 8, omission/supplement rules 12, and conjugation table 13, and the estimated words are supplemented 
s to the words having the dependency relationship to generate a spoken language sentence 1 4 and store it in the memory 
The details of Step 6 will be given later with reference to Fig. 6. 

[0031 ] The details of the word recognition procedure shown in Fig. 3 will be described. First, the principle of the word 
recognition method used in this embodiment will be described. In a sign language, a word is expressed by a series of 
motions of hands or fingers, and a sentence is expressed by a train of words. In this embodiment, the history data of 

10 a series of motions of hands, corresponding to a train of words, is stored as the inputted sign data 7. The word recog- 
nition program 3 compares, on the basis of maximum coincidence, i.e., a selection method of a train of words having 
the longest matching portion, the inputted sign data 7 with history data (not shown) of motions of hands learned in 
advance through a neural network. The train of words are cut out while being recognized. A typical learning and rec- 
ognition by a neural network is well known in the art. For example, refer to "Neural Computer - Learning from Brains 

is and Neurons", by Kazuyuki AIHARA, the Publication Department of Tokyo Electric College, 1988, pp. 93 to 128. 

[0032] At Step 31 , an initial value of a recognition section for the inputted sign data 7 are set. As the initial value, a 
start time of the recognition section P1 is set to time TO, and an end time of the recognition section P2 is set to time 
TO + W2. W2 is a maximum range value of the recognition section. 

[0033] At Step 32, data in the recognition section is taken out of the inputted sign data 7, and is inputted to an already 
20 learnt neural network. The neural network outputs recognized words Y (P1 , P2) and coincidence values Z (P1 , P2). 
The more a pattern of inputted data matches an already learnt data pattern, the larger the coincidence value Z is. If 
there are a plurality of words having the same coincidence value Z, these words may be outputted as a plurality of 
candidate words. Alternatively, a different evaluation method may be used, or one of a plurality of candidates may be 
preferentially selected in accordance with a predetermined order, e.g., the first processed word. 
25 [0034] At Step 33, the end time of the recognition section is shortened by one time. 

[0035] At Step 34, it is checked if the recognition section becomes smaller than the minimum value W1. If not, the 
procedure returns to Step 32 to again recognize data in the one-time shortened recognition section. 
[0036] At Step 35, a word having the most accurate recognizing result is selected from among recognized words Y 
(P1, P2) having the maximum coincidence values Z (P1 , P2) in a case of the end time in range of W1 to W2, and is 
30 stored in the analysis table 8 at the word field 81 . 

[0037] At Step 36, the recognition section is changed to the next recognition section, and W2 is set as an initial value 
of the start point for the next recognition section. 

[0038] At Step 37, it is checked whether the whole inputted sign data 7 has been processed. If not, the procedure 
returns to Step 32 to continue the word recognition. If so, the word recognition is terminated. In this manner, a train of 

35 words are recognized from the inputted sign data 7 and stored in the analysis table 8. 

[0039] As another method or the word recognition procedure, a dynamic programming (DP) matching scheme well 
known in the field of speech recognition, or other methods, may also be used. For example, a continuous DP matching 
scheme has been proposed by OKA in "Continuous Word Recognition using Continuous DP", the Speech Study Group 
of Acoustical Society of Japan, S78-20, pp. 145 to 152, 1978. By using this or other techniques, it is possible to dy- 

40 namically check the matching between a time sequential and continuous pattern and reference patterns. According to 
the above-cited document, it is recognized whether a reference pattern is included in a continuous pattern, through 
the continuous matching while the reference pattern is moved in a direction of the continuous pattern. During this 
matching, the time sequence of similarity factors of the continuous patterns relative to the reference patterns are ob- 
tained. A portion of the continuous pattern having a minimum of the similarity factors which is equal to or lower than a 

4 $ threshold value is selected as a candidate for the reference pattern. 

[0040] However, if data sampled at a predetermined time interval is used as it is for the continuous pattern and the 
reference patterns, the time required for pattern matching becomes long in proportion to the length of continuous 
pattern and the number of reference patterns. 

[0041] Effective word recognition of the sign language can be performed in the following manner. Specifically, in 
so order to normalize a sign language pattern considering non-linear expansion/compression, a corresponding relation 
is taken between samples of the reference pattern through DP matching, an average of which samples is determined 
to produce a sign language reference pattern. In the continuous sign language pattern recognition using the continuous 
DP matching, the continuous sign language pattern and a reference pattern are compressed and checked whether 
both coincide with each other, i.e., the matching is obtained while allowing the non-linear expansion/compression in 
55 the time domain. 

[0042] In a speech of the sign language, it is often that persons or articles are expressed by assigning them particular 
positions, and each position is used thereafter as the pronoun of each person or article. For example, assuming that 
a person A positions at the front right position and a person B positions at a front left position, a motion of " &t " = 
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"hanasu" = "speaking" from person A to person B, means "person A speaks to person B". Since the word recognition 
at Step 3 supplies the position information, it is possible to correctly recognize a person or article indicated by the 
pronoun represented by the position information used in the preceding speech. 

[0043] Fig. 4 shows the contents of the operation to be executed by the dependence analysis program 4. In the 
following description, a train of words 88 shown in Fig. 8 are illustratively used. This word train has five words V " = 
"watakushi" = T. "* " = "fuyu" = "winter", " " = " Hokkaido", " n< " = "iku" = "go", and " = "omou" = "think". 
[0044] At Step 41 , the words in the word field 81 of the analysis table 8 are checked sequentially starting from the 
start word, whether each word matches to a word of the word dictionary 9. The part of speech 92 and meaning 93 in 
the word dictionary for the matched word are entered in the part of speech field 82 and meaning field 83 of the analysis 
table 8. 

[0045] At Step 42, the depended word and dependence relationship of each matched word are analyzed and entered 
in the depended word field 84 and dependence relationship field 85 of the analysis table 8. 

[0046] The details of Step 42 will be given with reference to Fig. 5. First, the principle of the dependence relationship 
analysis will be described. It is known that a dependence relationship is present between phrases constituting a sen- 
tence, and a one-sentence /one-case principle and a no-cross principle are applied. 

[0047] The one -sentence/one-case principle teaches that a different case is provided to each predicate in one sen- 
tence, without occurrence of the same two cases. The no^ross principle teaches that a line indicating the dependence 
relationship will not cross another line. The dependence analysis for a train of words is performed using these principles. 
[0048] At Step 51 , the words 81 in the analysis table 8 are sequentially read from the analysis table 8, starting from 
the first word. 

[0049] At Step 52, it is checked if there is no word to be processed. If there is no word, the procedure is terminated. 
[0050] At Step 53, the read-out words are stacked on the analysis stack area 10. Fig. 10 shows the four words 
stacked, staring from the first word. 

[0051] At Step 54, the part of speech of each word is checked. If the part of speech is a predicate such as a verb, 
the procedure advances to Step 55, and if not, returns to Step 51 . 

[0052] At Step 55, the case dictionary 11 is referred to with respect to the verb on the analysis stack area 10 to 
determine the cases of depending words, through comparison between the contents of the case dictionary 11 and 
analysis stack area 10. In the example shown in Fig. 10, the verb " < " = "iku" = "go" is referred to the case dictionary 
1 1 to check the contents shown in Fig. 11 . Upon the comparison with the contents on the analysis stack area 10 shown 
in Fig. 10, a correspondence can be obtained, which shows that " ^ " = "watakushi" = "I" 1001 is the subjective case 
for the verb "«<• " = "iku" = "go", " *• " = "fuyu" = "winter" is the time case, and ■ «5fc& " = "Hokkaido" is the place case. 
This data is set in the analysis table 10 at the depended word field 84 and dependence relationship field 85. 
[0053] In the above example, the case frame of predicates has been described for the dependence relationship 
analysis. The dependence relationship between substantives such as nouns may also be performed using the case 
frame of substantives. For example, the noun frame " = ■ "dokokara" = "from where" and " ez&T " = "dokomade" 
= "to where" for defining a "#fc" = "kyori" = "distance" determine a certain dependence relationship between nouns. 
[0054] If the dependence relationship is ambiguous and cannot be analyzed definitely, the most possible one may 
be selected or a plurality of candidates may be used at the later processes. 

[0055] At Step 56, the entries on the analysis stack area 1 0 whose dependence relationships have been determined, 
are cleared with only the verb being left uncleared, the remaining words of the train are set on the analysis stack 10 
to further continue the dependence analysis. 

[0056] As understood from the above description, when the next word ■ t& " = "omou" = "think" is set on the analysis 
stack area 10, the contents of "V? " = "omou" = "think" in the case dictionary 11 are referred to and compared with the 
contents in the analysis stack area 10. As a result, the word " tt< M « "iku" = "go" is determined as the objective case 
of the word "omou" = "think". The results of the dependence analysis are stored in the analysis table 8. It is 

noted that there is no-cross between dependence relationships in the above analysis. 

[0057] Fig. 6 shows the details of the operation to be executed by the omission/supplement program 6. This program 

supplements omitted words by using the omission/supplement rules 12 or the like, while considering the word pairs 

having a dependence relationship determined by the dependence analysis program 4. 

[0058] At Step 61 , the words 81 in the analysis table 8 are sequentially read starting from the first word. 

[0059] At Step 62, it is checked whether a pair of depending and depended words, and the dependence relationship 

85 between the words match to conditions of a proper omission/supplement rule 12 to search for the rule satisfying 

the conditions. Next, the supplement method designated by the searched rule is executed and the results are stored 

in the analysis table 8 at the supplement result field 86. 

[0060] At Step 63, it is checked if all the words in the analysis table 8 have been processed. If not, the procedure 
returns to Step 61 to repeat the above operation. 

[0061] The above procedure will be explained by using a particular example. As the first word ' *k " = "watakushi" = 
"I" is read from the analysis table 8, the depended word 1 tt< " = "iku" = "go" and dependence relationship "***" = 
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"shukaku" = "subjective case" are also read, and these are compared with each of the omission/supplement rules 12 
to identify the rule 121 . The supplement method 123 is therefore executed so that the case postposition " u" = "wa" 
(Japanese postpositional particle representing the subjective case) is read from the case dictionary 11 and supple- 
mented to the word " " = "watakushi" = "I" after it. The results ' *m* " = "I am" are stored in the analysis table 8 at the 

s supplement result field 86. Similarly, the case postpositions "c " = "ni" = in" and ' = "e" = "to" are supplemented to 
the words = "fuyu" = "winter" and " = "Hokkaido" = "hokkaido", respectively, to obtain the results B *i* " = 

"fuyu ni" = "in the winter" and Mti«HL~ " = "Hokkaido e" "to Hokkaido". To the word pair of " «r<" = "iku" = "go" and 'tsb " 
= "omou" = "think", the rule 121 is first applied, and the case postposition m t " = "to" = "of" is supplemented after the 
word " «< " = "iku" = "go". Next, the rule 124 is applied, and the supplement method 126 is executed so that ■ * ■ = V 

10 (Japanese postpositional particle indicating intention) is supplemented between "«< ■ = "jku" = "go" and " t " = "to" 
= "of". The conjugations of this word "« < " = "iku" = "go" are further determined so as to satisfy the conjugative conditions 
defined by the supplement method 126. Accordingly, the "*£#> ," = "mizenkei" = "negative form (this is an arbitrary 
term since "mizenkei" has various functions according to the Japanese grammar)" 2 of the word B tt< " = "jku" = "go" 
is determined as " = "ko" 131 from the conjugation table 13. Similarly, the conclusive form of "j "= "u" is determined 

15 as " 6 " - "u" 132 from the conjugation table 13. In this manner, the result obtained by the supplement method 126 
becomes " tt^ * " = "ikou to" = "of going" which is stored in the analysis table 8 at the supplement result field 86. 
[0062] If any omission/supplement rule cannot be found, the word 81 per se is stored in the supplement result field 
86. In this case, even if the analysis fails in the midst of procedures, the train of words analyzed until then are outputted, 
to help the user obtain some information. A symbol representing a fail of supplement may be stored in the supplement 

20 result field 86. 

[0063] At Step 64, the supplement results stored in the analysis table at the result field 86 are connected together 
to generate and store a translated spoken-language sentence 14 which is displayed on CRT 103 as a translated 
spoken-language sentence 107. The train of words 81 entered as the sign words 106 are also displayed on CRT 103. 
The translated spoken-language sentence may be outputted as sounds by using a voice synthesizer. 

25 [0064] The motions of hands orthe like may be displayed on CRT 103 in response to the translated spoken-language 
sentence by using computer graphics techniques. In such a case, the procedures of this embodiment are reversed by 
time sequentially synthesizing images while referring to a dictionary of sign elements each representing a minimum 
unit of a sign language word, and to a dictionary of words each constituted by a combination of sign elements. With 
synthesized images of a sign language, an auditory-impaired person can visually confirm his or her sign language. 

30 The output may be any combination of a sentence text, voice representations, and synthesized images. 

[0065] According to the present invention, omitted words between the sign language words can be estimated and 
supplemented, allowing to translate a sign language into a spoken language. 

[0066] The present invention is not limited only to a Japanese sign language, but is applicable to various other sign 
languages. In the case of Japanese language, omitted words can be supplemented using case frame grammar models 
35 as in the above-described embodiment. For the European and American languages such as English, the order of words 
has a significant meaning. In such a case, omitted words may be supplemented by using syntactic grammar models. 



Claims 

40 

1. A sign language translation system, comprising: 

input means (105) for inputting the motion of hands as electric signals; 

language generating means (102) for generating a spoken language from said inputs, including sign language 
45 word generating means (3) for recognizing words corresponding to hand motions in accordance with said 

inputs and generating sign language words; 

output means (103) for outputting said generated spoken language, 
characterised in that said language generating means (102) further includes: 

50 

storage means (12,13) for storing conjugations of those of said sign language words which are conjugative 
words, and omitted words to be supplemented between said sign langauge words, said omitted words being 
words not corresponding to hand motions; 

dependence analyzing means (4) for analyzing a dependence relationship between successive ones of said 
55 recognized words and outputting analysed results; 

spoken language generating means (6) for generating said spoken language in accordance with said analysed 
results by providing said stored conjugations of said conjugative words and supplementing said stored omitted 
words between said recognized words. 
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2. A sign language translation system according to claim 1 , wherein said input means (1 05) comprises an input device 
for inputting the motion of the hands at predetermined time interval, 

3. A sign language translation system according to claim 1, wherein said sign language word generating means (3) 
s recognizes sign language words using a neural network which learns hand motions corresponding to said sign 

language words by sequentially performing matching comparison between said learned hand motions and said 
input motion of the hands. 

4. A sign language translation system according to claim 1 , wherein said sign language word generating means (3) 
10 recognizes sign language words by comparing input motions of hands with stored hand motions based on a DP 

matching scheme. 

5. A sign language translation system according to claim 1 , wherein when the sign language word generating means 
(3) is unable to identify a final single word, a plurality of candidate words are sent to said dependence analyzing 

is means (4). 

6. A sign language translation system according to claim 1, wherein the storage means further includes a word dic- 
tionary (9) and a case dictionary (11 ), and said dependence analyzing means (4) analyses the dependence rela- 
tionship by: 

20 

storing said recognised words in a stack memory (81) in the sequence of word recognition; 
producing an analysis table (8) containing cases, parts of speech (182) and meaning of said stacked words 
using the word dictionary (9) and the case dictionary (11); 
determining a word to be analysed in the stack; 
25 searching words from the stack which are capable of depending on said determined word in accordance with 

said analysis table; and 

storing said searched words into a memory (86). 



7. A sign language translation system according to claim 6, wherein said dependence analysing means determines 
30 a word to be analysed in the stack when the word is a declinable word, wherein a declinable word indicates a word 

describing an action, function, existence, property or condition thereof. 

8. A sign language translation system according to claim 7, wherein said dependence analyzing means moves said 
searched words corresponding to said determined word from the stack to said analysis table together with infor- 
ms mation concerning the dependence relationship (85) between the determined word and the searched words, where- 
in said determined word remains in the stack. 

9. A sign language translation system according to claim 8, wherein said dependence analyzing means further per- 
forms dependence analysis with respect to substantives. 

40 

10. A sign language translation system according to claim 9, wherein said spoken language generating means sup- 
plements said stored postpositions or auxiliary verbs in accordance with said moved words in said analysing table 
and corresponding information concerning a dependence relationship between the determined words and the 
moved words. 

45 

11. A sign language translation system according to claim 1, wherein said omitted words to be supplemented are 
selected from an auxiliary verb, postposition, pronoun, conjunction, respectively representing a semantic and se- 
quential relationship between words. 

50 12. A sign language translation system according to claim 1, wherein said output means (103) outputs the train of 
spoken language words corresponding to said generated sign language words recognized when a spoken lan- 
guage cannot be generated. 



55 



13. A sign language translation system according to claim 1, wherein said language generating means (102) further 
comprises: 

word reading means for reading the words recognised by said sign language word generating means (3); 
analysis stacking means (10) for individually stacking the read words comprising a sentence; 
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speech determining means for determining the part of speech of each word stacked in said analysis stacking 
means; 

a case dictionary (11) for determining the conjugation of each word in said analysis stacking means relative 
to the verb determined by said determining means; and 

an analysis table (14) for receiving the properly conjugated words from said analysis stacking means (10) 
relative to the verb. 

14. A sign language translation system according to claim 1, wherein when said dependence analysing means fails 
to analyse a dependence relationship between said determined words and said searched words, said spoken 
language generating means outputs recognized words without supplementing said stored omitted words and with- 
out providing said stored conjugations of conjugative words. 

15. A sign language translation method comprising the steps of: 

inputting the motion of hands; 

recognizing words corresponding to said inputted motion of hands as sign language words; 
generating a spoken language using the dependence between said recognised words; and 
outputting said generated spoken language; 

characterised in that: 

conjugations of those of said sign language words which are conjugative words, and omitted words to be 
supplemented between said sign langauge words, said omitted words being words not corresponding to hand 
motions are stored; 

the dependence between said recognised words is analysed, and a predetermined process is executed using 
said analysed results to generate said spoken language by supplementing said omitted words and providing 
said conjugations of words that require conjugation. 

16. A sign language translation method according to claim 1 5, wherein spatial information representing a pronoun and 
time lapse entered in a sign language is recognized to translate said spatial information into a corresponding word. 

Patentanspruche 

1. Ubersetzungssystem fur Gebardensprache, umfassend: 

Eingabemittel (105) zum Eingeben der Bewegung von Handen als elektrische Signale; Spracherzeugungs- 
mittel (102) zum Erzeugen einer gesprochenen Sprache aus den Eingaben, umfassend Gebardensprachen- 
Worterzeugungsmittel (3) zum Erkennen von Wortern, die Handbewegungen entsprechen, gemaB den Ein- 
gaben, und Erzeugen von Gebardensprachewortern; 
Ausgabemittel (103) zum Ausgeben der erzeugten gesprochenen Sprache, 

dadurch gekennzeichnet, daB das Spracherzeugungsmittel (102) weiters umfaBt: 

Speichermittel (12, 13) zum Speichern von Konjugationen von jenen der Gebardensprache wo rter, die konju- 
gierbare Worter sind, und weggelassenen Wortern, die zwischen den Gebardensprachewortern zu erganzen 
sind, wobei die weggelassenen Worter solche sind, die keinen Handbewegungen entsprechen; 
Abhangigkeitsanalysemittel (4) zum Analysieren einer Abhangigkeitsbeziehung zwischen aufeinanderfolgen- 
den der erkannten Worter und Ausgeben von Analyseergebnissen; 

Erzeugungsmittel (6) fur gesprochene Sprache zum Erzeugen der gesprochenen Sprache gemaB den Ana- 
lyseergebnissen durch Bereitstellen der gespeicherten Konjugationen der konjugierbaren Worter und Ergan- 
zen der gespeicherten weggelassenen Worter zwischen den erkannten Wortern. 

2. Ubersetzungssystem fur Gebardensprache nach Anspruch 2, worin das Eingabemittel (105) eine Eingabevorrich- 
tung zum Eingeben der Bewegung der Hande in vorbestimmtem Zeitintervall umfaBt. 

3. Ubersetzungssystem fur Gebardensprache nach Anspruch 1 , worin das Gebardensprache-Worterzeugungsmittel 
(3) Gebardenspracheworter unter Verwendung eines neuronalen Netzes erkennt, das Handbewegungen, die den 
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Gebardensprachewortern entsprechen, durch aufeinanderfolgende Durchfuhrung von Gleichheitsprufungsver- 
gleich zwischen den gelernten Handbewegungen und der eingegebenen Bewegung der Hande lernt. 

4. Ubersetzungssystem fur Gebardensprache nach Anspruch 1 , worin das Gebardensprache-Worterzeugungsmittel 
s (3) Gebardenspracheworter erkennt, indem es eingegebene Bewegungen von Handen auf Basis eines DV-Gleich- 

heitsprOfungsschemas mil gespeicherten Handbewegungen vergleicht. 

5. Ubersetzungssystem fur Gebardensprache nach Anspruch 1, worin, wenn das Gebardensprache-Worterzeu- 
gungsmittel (3) nicht in der Lage ist, ein endgultiges einzelnes Wort zu identifizieren, eine Vielzahl von Kandida- 

10 tenwortern an das Abhangigkeitsanalysemittel (4) geschickt wird. 

6. Ubersetzungssystem fur Gebardensprache nach Anspruch 1 , worin das Speichermittel weiters ein Begriffsworter- 
buch (9) und ein Fallwdrterbuch (11) umfaGt und das. Abhangigkeitsanalysemittel (4) die Abhangigkeitsbeziehung 
analysiert durch: 

is 

Speichern der erkannten Worter in einem Stapelspeicher (81 ) in der Reihenfolge der Worterkennung; 

Erstellen einer Analysetabelle (8), die Falle, Sprachenteile (182) und die Bedeutung der gestapelten Worter 
enthalt, unter Verwendung des Begriffsworterbuchs (9) und des Fallwdrterbuchs (11); 

20 

Bestimmen eines zu analysierenden Wortes im Stapel; 

Suchen nach Wortern aus dem Stapel, die vom bestimmten Wort abhangen konnen, gemaB der Analyseta- 
belle; und 

Speichern der gesuchten Worter in einem Speicher (86). 

7. Ubersetzungssystem fur Gebardensprache nach Anspruch 6, worin das Abhangigkeitsanalysemittel ein zu ana- 
lysierendes Wort im Stapel bestimmt, wenn das Wort ein deklinierbares Wort ist, wobei mit deklinierbares Wort 
ein Wort gemeint ist, das eine Handlung, eine Funktion, ein Vorhandensein, eine Eigenschaft oder einen Zustand 
davon beschreibt. 

8. Ubersetzungssystem fur Gebardensprache nach Anspruch 7, worin das Abhangigkeitsanalysemittel die gesuchten 
Worter, die dem bestimmten Wort entsprechen, gemeinsam mit Informationen uber die Abhangigkeitsbeziehung 
(85) zwischen dem bestimmten Wort und den gesuchten Wortern aus dem Stapel zur Analysetabelle bewegt, 
wobei das bestimmte Wort im Stapel verbleibt. 

9. Ubersetzungssystem fur Gebardensprache nach Anspruch 8, worin das Abhangigkeitsanalysemittel weiters Ab- 
hangigkeitsanalyse in bezug auf Substantive durchfuhrt. 

10. Ubersetzungssystem fur Gebardensprache nach Anspruch 9, worin das Erzeugungsmittel fur gesprochene Spra- 
che die gespeicherten Postposition en oder Auxiliarverben gemaG der bewegten Worter in der Analysetabelle und 
entsprechender Informationen bezOglich einer Abhangigkeitsbeziehung zwischen den bestimmten Wortern und 
den bewegten Wortern erganzt 

11. Ubersetzungssystem fur Gebardensprache nach Anspruch 1, worin die zu erganzenden weggelassenen Worter 
aus einem Auxiliarverb, einer Postposition, einem Pronomen, einer Konjunktion ausgewahlt sind, die jeweils eine 
semantische und sequentielle Beziehung zwischen Wortern darstellen. 

12. Ubersetzungssystem fur Gebardensprache nach Anspruch 1 , worin das Ausgabemittel (1 03) die Abfolge von Wor- 
tern gesprochener Sprache ausgibt, die den erzeugten Gebardensprachen-Wortern entsprechen, die erkannt wur- 
den, wenn eine gesprochene Sprache nicht erzeugt werden kann. 

13. Ubersetzungssystem tur Gebardensprache nach Anspruch 1, worin das Spracherzeugungsmittel (102) weiters 
55 umfaGt: 

Wortlesemittel zum Lesen der vom Gebardensprachen-Worterzeugungsmittel (3) erkannten Worter; 
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Analysestapelmittel (10) zum individuellen Stapeln der geiesenen Worter, die einen Satz umfassen; 

Sprachbestimmungsmittel zum Bestimmen des Sprachenteils eines jeden im Analysestapelmittel gestapelten 
Wortes; 

ein Fallworterbuch (11 ) zum Bestimmen der Konjugation eines jeden Wortes im Analysestapelmittel in bezug 
auf das Verb, das durch das Bestimmungsmittel bestimmt ist; und 

eine Analysetabelle (14), urn die in bezug auf das Verb richtig konjugierten Worter vom Analysestapelmittel 
(10) zu erhalten. 

14. Ubersetzungssystem fur Gebardensprache nach Anspruch 1, worin, wenn es dem Abhangigkeitsanalysemittel 
nicht gelingt, eine Abhangigkeitsbeziehung zwischen den bestimmten Wortern und den gesuchten Wortern her- 
zustellen, das Erzeugungsmittel fur gesprochene Sprache erkannte Worter abgibt, ohne die gespeicherten aus- 
gelassenen Worter zu erganzen und ohne die gespeicherten Konjugationen konjugierter Worter zu liefern. 

15. Ubersetzungsverfahren fur Gebardensprache, folgende Schritte umfassend: 

das Eingeben der Bewegung von Handen; 

das Erkennen von Wortern, die der eingegebenen Bewegung von Handen entsprechen, als Worter der Ge- 
bardensprache; 

das Erzeugen einer gesprochenen Sprache unter Verwendung der Abhangigkeit zwischen den erkannten 
Wortern, und das Ausgeben der erzeugten gesprochenen Sprache; 

dadurch gekennzeichnet, daB 

Konjugationen jener der Worter der Gebardensprache, bei denen es sich urn konjugierbare Worter handelt, 
sowie weggelassene Worter, die zwischen den Wortern der Gebardensprache zu erganzen sind, gespeichert 
werden, wobei die weggelassenen Worter solche sind, die nicht Handbewegungen entsprechen; 
die Abhangigkeit zwischen den erkannten Wortern analysiert wird und unter Einsatz der Analyseergebnisse 
ein vorbestimmtes Verfahren durchgefuhrt wird, um die gesprochene Sprache zu erzeugen, indem die weg- 
gelassenen Worter erganzt und die Konjugationen von Wortern bereitgestellt werden, die Konjugation erfor- 
dern. 

16. Ubersetzungsverfahren fur Gebardensprache nach Anspruch 15, worin raumliche Information, die fur ein Prono- 
men stent, und verstrichene Zeit, die in eine Gebardensprache Aufnahme findet, erkannt wird, um die raumliche 
Information in ein entsprechendes Wort zu ubersetzen. 



Revendications 

1. Systeme de traduction de langage des signes, comprenant : 

un moyen d'entree (105) pour entrer le mouvement de mains comme signaux electriques ; 

un moyen de production de langage ( 1 02) pour produire un langage parte a partir desdites entrees, comprenant 

un moyen de production de mots de langage des signes (3) pour reconnaitre des mots correspondant a des 

mouvements de mains selon lesdites entrees et produire des mots de langage des signes ; 

un moyen de sortie (103) pour produire ledit langage parle produit, 

caracterise en ce que ledit moyen de production de langage (102) comprend en outre : 

un moyen de memorisation (12, 13) pour memoriser des conjugaisons desdits mots de langage des signes 
qui sont des mots conjugatifs, et des mots omis a ajouter entre lesdits mots de langage des signes, lesdits 
mots omis etant des mots ne correspondant pas a des mouvements de mains ; 

un moyen d'analyse de dependance (4) pour analyser une relation de dependance entre ceux successifs 
desdits mots reconnus et produire et des r^sultats analyses; 

un moyen de production des langage parle (6) pour produire ledit langage parle selon lesdits resultats analyses 
en foumissant lesdites conjugaisons memorisees desdits mots conjugatifs et ajoutant lesdits mots omis me- 
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morises entre lesdits mots reconnus. 

2. Systeme de traduction de langage des signes selon la revendication 1 , dans lequel !e moyen d'entree precite (1 05) 
comprend un dispositif d'entree pour entrer le mouvement des mains a un intervalle de temps predetermine. 

3. Systeme de traduction de langage des signes selon la revendication 1, dans lequel le moyen de production de 
mots de langage des signes precite (3) reconnaTt des mots de langage des signes utilisant un reseau neural qui 
apprend des mouvements de mains correspondant auxdits mots de langage des signes en accomplissant sequen- 
tiellement une comparaison d'accord entre lesdits mouvements des mains appris et ledit mouvement d'entree des 

10 mains. 

4. Systeme de traduction de langage des signes selon la revendication 1 , dans lequel le moyen de production de 
mots de langage des signes precite (3) reconnaTt des mots de langage des signes en comparant des mouvements 
d'entree des mains a des mouvements de mains memorises sur la base d'un schema d'accord DP. 

15 

5. Systeme de traduction de langage des signes selon la revendication 1 , dans lequel lorsque le moyen de production 
de mots de langage des signes precite (3) est incapable d'identifier un mot unique final, une pluralite de mots 
candidats sont transmis au moyen d'analyse de dependance precite (4). 

20 6. Systeme de traduction de langage des signes selon la revendication 1, dans lequel le moyen de memorisation 
comprend en outre un dictionnaire de mots (9) et un dictionnaire de cas (11 ), et le moyen d'analyse de dependance 
precite (4) analyse la relation de dependance en : 

memorisant les mots reconnus precites dans une memoire a registre (81 ) dans la sequence de reconnaissance 
25 des mots ; 

produire un tableau d'analyse (8) contenant des cases, des parties de paroles (182) et la signification desdits 
mots memorises en utilisant le dictionnaire de mots (9) et ie dictionnaire de cas (11). 
determinant un mot a analyser dans le registre ; 

recherchant des mots a partir du registre qui sont capables de dependre dudit mot determine selon ledit tableau 
30 d'analyse ; et 

memorisant lesdits mots recherches dans une m6moire (86). 

7. Systeme de traduction de langage des signes selon la revendication 6, dans lequel le moyen d'analyse de depen- 
dance precite determine un mot a analyser dans le registre lorsque le mot est un mot declinable, ou un mot de- 

35 clinable indique un mot decrivant une action, une fonction, une existence, une propriete ou une condition de celui-ci. 

8. Systeme de traduction de langage des signes selon la revendication 7, dans lequel le moyen d'analyse de depen- 
dance precite deplace les mots recherches precites correspondant au mot determine precite du registre au tableau 
d'analyse precite ensemble avec une information concernant la relation de dependance (85) entre le mot determine 

40 et les mots recherches, ou ledit mot determine reste dans le registre. 

9. Systeme de traduction de langage des signes selon la revendication 8, dans lequel le moyen d'analyse de depen- 
dance precite accomplit en outre une analyse de dependance par rapport a des substantifs. 
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10. Systeme de traduction de langage des signes selon la revendication 9, ou le moyen de production de langage 
parle precite ajoute des postpositions ou des verbes auxiliaires memorises precites selon les mots deplaces pre- 
cites dans le tableau d'analyse precite et une information correspondante concernant une relation de dependance 
entre les mots determines et les mots deplaces. 

1 1 . Systeme de traduction de langage des signes selon la revendication 1 , dans lequel les mots omis precites a ajouter 
sont selectionnes a partir d'un verbe auxiliaire, d'une postposition, d'un pronom, d'une conjonction, representant 
respectivement une relation semantique et sequentielle entre des mots. 

12. Systeme de traduction de langage des signes selon la revendication 1, dans lequel le moyen de sortie precite 
(103) produit le train de mots de langage parle correspondant aux mots de langage des signes produits precite 
reconnus lorsqu'un langage parle ne peut pas etre produit. 

13. Systeme de traduction de langage des signes selon la revendication 1, dans lequel le moyen de production de 
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langage precite (102) comprend en outre : 

un moyen de lecture de mots pour lire les mots reconnus par le moyen de production de mots de langage des 
signes precite (3) ; 

un moyen de memorisation d'analyse (10) pour memoriser individuellement les mots lus comprenant une 
phrase ; 

un moyen de determination de parole pour determiner la partie de parole de chaque mot memorise dans le 
moyen de memorisation et d'analyse; 

un dictionnaire de cas (11) pour determiner la conjugaison de chaque mot dans ledit moyen de memoire 
d'analyse par rapport au verbe determine par ledit moyen de determination ; et 

un tableau d'analyse (14) pour recevoir les mots correctement conjugues dudit moyen de memoire d'analyse 
(10) par rapport au verbe. 

14. Systeme de traduction de langage des signes selon la revendication 1, dans lequel lorsque le moyen d'analyse 
de dependance precite fait defaut pour analyser une relation de dependance entre les mots determines precites 
et les mots recherches precites, le moyen de production de langage parie precite produit des mots reconnus sans 
ajouter les mots omis memorises precites et sans fournir les conjugaisons memorisees precitees de mots conju- 
gatifs. 

15. Precede de traduction de langage des signes comprenant les etapes de : 

entrer le mouvement de mains ; 

reconnattre des mots correspondant audit mouvement entre de mains comme mots de langage des signes ; 
produire un langage parie en utilisant la dependance entre lesdits mots reconnus ; et 
produire ledit langage parie produit ; 

caracterise en ce que : 

des conjugaisons des mots desdits mots de langage des signes qui sont des mots pouvant etre conjugues et 
des mots omis a ajouter entre lesdits mots de langage des signes, lesdits mots omis etant des mots ne cor- 
respondant pas a des mouvements de mains, sont memorises ; 

la dependance entre lesdits mots reconnus est analysee, et un procede predetermine est execute utilisant 
lesdits resultats analyses pour produire ledit langage parie en ajoutant lesdits mots omis et fournissant lesdites 
conjugaisons de mots qui exigent une conjugaison. 



16. Procede de traduction de langage des signes selon la revendication 15, dans lequel une information spatiale 
presentant un pronom et.un ecoulement de temps entres dans un langage des signes est reconnue pour traduire 
ladite information spatiale en un mot correspondant. 
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